It often is desirable to automate many operations currently performed by humans using a robotic device. For example, the operations may be repetitive, dangerous, and/or the like. However, successful automation using a robotic device requires that the robotic device have an understanding of the environment within which the robotic device is tasked with performing the operations.
A railyard is an illustrative example of an environment, in which is desirable to automate various operations generally performed by humans. For example, an individual present in the railyard will manually activate a cut lever (e.g., a coupler release handle) in order to separate two rail vehicles, manually activate a brake rod (e.g., an air system release lever) to bleed the brakes of a rail vehicle, and/or the like. In terms of the basic capability, any individual capable of physically travelling along the tracks in the railyard and exerting the required force will have no problem performing these relatively straightforward operations. However, humans in these settings are quite expensive, costing $50/hour or more. More importantly, such operations in an environment where the rail vehicles may be moving is quite hazardous. A single misstep can cause anything from bruises to death. Humans also tire relatively easily and cannot always keep track of all events that may occur while performing what are, mostly, boring and repetitive tasks.
One approach proposes an automated, stationary pin-puller for railcars, which recognizes the target pin and associated components and then performs a pin-pull. However, the discussion does not address various issues associated with real world applications, including recognizing the target in multiple confounding circumstances, movement of rail vehicles at varying speeds, and/or the like. For example, a rail vehicle moving a higher than normal rate of speed will not permit the pin to be successfully pulled in the short window of time between the pin and associated hardware becoming visible to the system and the following car moving into that space. As a result, a collision of the pin-pulling arm with the following car could result, likely severely damaging or destroying the automated pin-puller. Furthermore, a location at which the pin can be pulled can vary based on the length and load on the rail vehicle being separated, which can present a challenge in implementing a fixed location pin-puller.
An illustrative example of a robotic device which can be used to automate or semi-automate rail operations is shown and described in U.S. Patent Application Publication No. 2010/0076631, which is hereby incorporated by reference. As described therein, the robotic device can be configured to operate in a railyard, and be tasked with various operations relating to the processing of rail vehicles on consists located in the railyard at particular locations. For example, the robotic device can be configured to operate autonomously or semi-autonomously to activate a cut lever in order to separate two rail vehicles, activate a brake rod to bleed the brakes of a rail vehicle, and/or the like. Successful operation by the robotic device requires an ability of the robotic device to recognize individual rail vehicles, locate the cut lever and/or brake rod, and activate the cut lever or brake rod as needed, while distinguishing the real targets from multiple false targets. Additionally, the robotic device generally requires an understanding of the attributes of the scene, such as the forces acting on the rail vehicles.
A robotic device moving alongside a track is uniquely challenged in trying to recognize small targets. Unlike human beings, who have an immense amount of evolved, dedicated biological hardware and instinct which are focused around understanding what they see in their environment, even the best robotic devices do not actually “understand” what their cameras “see”. A robotic device must carefully analyze a scene using various heuristics and algorithms and determine whether a given set of characteristics is present. Unfortunately, for many potential targets, the simple target characteristics—loops, straight lines, etc.—which can be derived by a reasonable-sized computational platform in real time can result in many false positives.
For example, a robotic device attempting to locate a brake rod that ends in a loop may identify multiple loops in image data acquired by the cameras, including those from lettering or shadows, and can have a significant challenge in trying to determine which of these loops is, in fact, the brake rod, or whether the robotic device is even in the correct location to locate the brake rod. This is further exacerbated since a moving robotic device will be changing its angle of view in three dimensions as it moves, may go over bumps or into dips of the landscape, or can vibrate going over rough terrain. As a result, the robotic device can have difficulty in determining a precise location or bearing of the target.
A robotic device could be configured to implement more complex methods to reduce or eliminate uncertainty in analyzing the environment. However, such methods require considerable computation time, which can make real time implementation on a mobile platform, whose mobility itself introduces considerable additional difficulty to the problem, impractical or impossible.
Some approaches seek to perform major computations in real time using a stationary processing system located some reasonable distance from the operating robotic device. Such an approach reduces a need for carrying the computational capability onboard the robotic device, but still requires an extremely powerful system to perform in real-time, and introduces an additional major challenge of bandwidth for the communications between the robotic device and the stationary system. For example, in order to produce good results, the station system should have the same data available as the robotic device. In this case, all of the data gathered by the robotic device's vision system must be transmitted to the stationary system. For good resolution and accuracy in field conditions, reasonably high-resolution cameras must be used, and for a moving vehicle, the frame rate should be at least thirty frames per second and perhaps higher. For a full-color camera, even with some compression, this translates to tens of megabytes per second over a wireless channel, or hundreds of megabits per second (Mbps). Common wireless methods such as “WiFi” (802.11 standards) can barely exceed 100 Mbps in current incarnations. As a result, use of such an approach in commercial settings is expensive and in many cases completely impractical.